Prediction of highly cited papers

In an article in the pages of this journal five years ago, we described a method for predicting which scientific papers will be highly cited in the future, even if they are currently not highly cited. Applying the method to real citation data we made predictions about papers we believed would end up being well cited. Here we revisit those predictions, five years on, to see how well we did. Among the over 2000 papers in our original data set, we examine the fifty that, by the measures of our previous study, were predicted to do best and we find that they have indeed received substantially more citations in the intervening years than other papers, even after controlling for the number of prior citations. On average these top fifty papers have received 23 times as many citations in the last five years as the average paper in the data set as a whole, and 15 times as many as the average paper in a randomly drawn control group that started out with the same number of citations. Applying our prediction technique to current data, we also make new predictions of papers that we believe will be well cited in the next few years. editor’s choice Copyright c © EPLA, 2014 Introduction. – Citations of scientific papers are considered to be an indicator of papers’ importance and relevance, and a simple count of the number of citations a paper receives is often used as a gauge of its impact. However, it is also widely believed that citations are affected by factors besides pure scientific content, including the journal a paper appears in, author name recognition, and social effects [1,2]. One substantial and well-documented effect is the so-called cumulative advantage or preferential attachment bias, under which papers that have received many citations in the past are expected to receive more in the future, independent of content. A simple mathematical model of this effect was proposed by Price [3], building on earlier work by Yule [4] and Simon [5], in which paper content is ignored completely and citation is determined solely by preferential attachment plus stochastic effects. Within this model, the expected number of citations a paper receives is a function only of its date of publication, measured from the start of the topic or body of literature in which the paper falls, and shows a strong “first-mover effect” under which the first-published papers are expected to receive many more citations on average than those that come after them. Indeed the variation in citation number as a function of publication date is normally far wider than the stochastic variation among papers published at the same time. In a previous paper [6] we compared the predictions of Price’s model against citation data for papers from several fields and found good agreement in some, though not all, cases. This suggests that pure citation numbers may not be a good indicator of paper impact, since much of their variation can be predicted from publication dates, without reference to paper content. Instead, therefore, we proposed an alternative measure of impact. We proposed that rather than looking for papers with high total citation counts, we should look for papers with counts higher than expected given their date of publication. Since publication date is measured from the start of a field or topic, and since different topics have different start dates, one should only use this method to compare papers within topics. The appropriate calculation is to count the citations a paper has received and compare that figure to the counts for other papers on the same topic that were published around the same time. In our work we used a simple z-score to perform the comparison: we calculate the mean number of citations and its standard deviation for papers published in a window close to the date of a paper of interest, then calculate the number of standard deviations by which that paper’s citation